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Survey  Objectives 

The  objectives  of  this  survey  are  to  characterize 

•  the  degree  to  which  software  practitioners  use  measurement 
when  conducting  their  work 

•  the  perceived  value  of  measurement 

•  approaches  that  are  used  to  guide  how  measures  are  defined 
and  used 

•  the  most  common  types  of  measures  used 
by  software  practitioners 
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Characteristics  of  the  Survey 

We  used  a  structured,  self-administered  questionnaire  that  was  available 
both  via  the  World  Wide  Web  and  in  paper  form. 

The  questionnaire  was  designed  to  be  short  (17  questions)  and  easy-to- 
complete  with  questions  phrased  in  close-ended  format.  Several 
questions  allowed  for  short  open-ended  responses. 

Stratified  random  sampling  was  used  to  select  candidate  respondents 
from  a  population  comprised  of  members  from  three  different 
subpopulations. 

Candidate  respondents  were  offered  incentives  to  participate  including 

•  platinum  membership  to  the  Software  Engineering  Information 
Repository  (SEIR)  that  provides  access  to  documents  otherwise 
unavailable  through  regular  membership 

•  early  access  to  the  survey  results 
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The  Population  Being  Studied 

The  population  that  we  would  have  liked  to  have  studied  is  the 
entire  existing  body  of  software  practitioners  in  the  world. 
However,  such  a  representative  database  was  unavailable  to  us. 

The  population  that  we  did  use  for  this  study  included  individuals 
who: 

1  were  entered  into  the  SEI  customer  relations  database 
during  2004-2005 

2  registered  to  gain  access  to  the  SEI’s  Software 
Engineering  Repository  (SEIR)  during  2004-2005 

3  became  an  SEI  Member  during  2004-2005 
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Sampling  Plan 


Calculated  for: 

confidence  of 

•  YlvivWyr  .V  \V|V 

Population 

Size 

Sample 

Size 

Adjusted 
Sample  Size 

Actual 

Subpopulation 

Sample  Size 

Customer  Relations 

6,398 

603 

2010 

1670 

SEI  Members 

1,242 

1,242 

951 

SEIR  registrants 

7,540 

. 

2040 

1539 

Total  15,180 

5,292 

4,160 

Adjusted  based  on  estimated 
30%  response  outcome. 


•  Invalid  email  addresses 

•  Non-responses 

•  Ineligible  respondents 
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Important  to  Remember  When 
Interpreting  Survey  Results 


Population  of  all  SEI 

software  practitioners  Survey  results  can  not  be  generalized  Members 


beyond  the  population  used  in  this  study. 
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Response  Outcome  Rates 


Standard  Definitions 

Final  Dispositions  of  Case  Codes 
and  Outcome  Rates  for  Surveys 


ROD  Telephone  Surveys 
In-Person  Household  Surveys 
Mail  Surveys  of  Specifically  Named  Persons 
Internet  Surveys  of  Specifically  Named  Persons 


2006  

THE  AMERICAN  ASSOCIATION  FOR  PUBLIC  OPINION  RESEARCH 


RRl  =  42.4% 


Counts  partial  interviews  as  respondents 


RR2  =  50.7% 


http://www.aapor.org/pdfs/standarddefs_4.pdf 
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Respondents  by  Continent 
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Describe  Your  Organization 


Custom  software  development 
In-house  or  proprietary  development  or  maintenance 

Other 

Defense  contractor 
Department  of  Defense  or  military  organization 

Commercial  shrink-wrap 
Other  government  contractor 
Other  government  agency 


1 893  Responses 


Frequency 
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Survey  Respondents 


Other 
Project  manager 
Engineer 
Executive  manager 
Program  manager 
Analyst 
Programmer 


33.5% 


634 


340 


5.9% 


152 


5.2% 


67 


0  100  200  300  400  500  600 

Frequency 

1895  Responses 


700 
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Approximate  Population 
Proportions 
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Number  of  Full  Time  Employees 


Frequency 

1 890  Responses 
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❖  Results 

•  Response  rates  and  outcome 

•  Were  subpopulations  different? 

•  Population  demographics 

Attitudes  and  beliefs  about  measurement  use 

How  are  you  involved  with  measurement? 

Are  purposes  for  measurement  understood? 

Does  measurement  help? 

Is  measurement  used  to  understand  product/service  quality? 
Documented  measurement  processes  ? 

Measurement  definitions  understood  and  consistent? 

Do  measurable  criteria  exist  for  products  and  services  ? 

Is  corrective  action  taken  when  thresholds  are  exceeded? 

•  Measurement  guidance  used 

•  Measures  that  are  reported 
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Involvement  With  Measurement 


Provider 


User 


290 


324 


►  32.4% 


Both  a  provider  and  a  user 


60.3% 


1142 


Other 


7.1% 


136 


Don’t  do  measurement. 

I  set  up  measurement  programs. 


0  200  400 


600  800 

Frequency 


1000  1200 


1892  Responses 


1400 
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Provides  (only)  or  Uses  (only) 


40.0% 


30.0% 


20.0% 


10.0% 


0.0% 


Exec 

289  Provider 
321  User 


Prog.  Project  Engineer  Analyst  Programmer  Other 

Manager  manager 
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Purpose  for  Measuring  Is  Understood 


Frequency 
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Purpose  for  Measuring  Is  1847  Responses 

Understood  Occasionally 


1 00% 


80% 


60% 


40% 


20% 


0% 


Frequently 


19% 


18% 


75% 


76% 


Program 

Manager 


26% 


66% 


17% 


22% 


74% 


35% 


■ 


26% 


Significant  differences  between 
management  and  staff. 


Executive  Project 
Manager 


Other  :|  An  Si  ysf ::::::::::  Program  ffi  eif: : : :  Engineer 
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Believe  That  Measurement  Helps 
(To  Some  Degree) 


Agree: 


92% 


Disagree:  2% 
N/A:  2% 
Not  sure:  4% 


1868  Responses 
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Measurement  Used  to  Understand 
Quality  of  Products  &  Services 


1852  Responses  Frequency 
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Documented  Process  for 
Collecting  Measurement  Data 


Frequently 

Occasionally 

Rarely 

Never 


269  14.5% 


I  don’t  know  is  1.0% 


N/A 


87  4.7% 


43  2.3% 


0 


200 


1852  Responses 


400  600 

Frequency 


800 


1000 
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Documented  Process  for 
Collecting  Measurement  Data 


100% 


80% 


60% 


40% 


20% 


0% 

Program  Other  Executive  Project  Analyst  Engineer  Programmer 
Manager  Manager 


32% 

Occasionally 
■  Frequently 

30% 

27% 

33% 

30% 

30% 

53% 

50% 

"T" 

45% 
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32% 
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Measurement  Definitions  Are 
Understood  &  Consistent 

N/A:  2% 
Disagree 

Not  sure:  4% 

1868  Responses 


©  2006  by  Carnegie  Mellon  University 


2006  State  of  SW  Measurement  Practice  -  25 


Carnegie  Mel  Ion 

Software  Engineering  Institute 


Measurable  Criteria  Exist  for 
Products  &  Services 


Frequently 
Occasionally 
Rarely 
Never 
I  don’t  know 


233  12.6% 

31  1 .7% 

30  1 .6% 


N/A 


48  2.6% 


0  200  400  600  800 

Frequency 

1852  Responses 


1000 
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Corrective  Action  Taken  When 
Measurement  Threshold  Exceeded 


Frequency 
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Action-Oriented  Response  to 
Measurement  Information 


Percent 


Program  Other  Executive  Project  Engineer  Analyst  Programmer 
Manager  Manager 


□  Measurable  criteria  established  (frequently) 

Corrective  action  taken  when  threshold  met  (frequently) 


©  2006  by  Carnegie  Mellon  University 


2006  State  of  SW  Measurement  Practice  -  28 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Presentation  Outline 


❖  Introduction 

•  Survey  objectives  &  approach 

•  The  population  being  studied 

•  Sampling  plan 

❖  Results 

•  Response  rates  and  outcome 

•  Were  subpopulations  different? 

•  Population  demographics 

•  Attitudes  and  beliefs  about  measurement  use 

•  Measurement  guidance  used 

•  Measures  that  are  reported 

❖  Summary  Observations 


©  2006  by  Carnegie  Mellon  University 


2006  State  of  SW  Measurement  Practice  -  29 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Measurement  Methods  Used 


Frequency 

1868  Responses 
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“Other”  Methods  Used 


Home-Grown 
No  Metrics  (comment) 
Proprietary 
Other 

Customized  from  Other 
CMM 

Standards  (IEEE,  ISO,  etc.) 

Six-Sigma 

ITIL 

Balanced  Scorecard 
By  Mandate 
SPC 

Function  Points 


Frequency 
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❖  Introduction 
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•  Response  rates  and  outcome 

•  Were  subpopulations  different? 
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•  Measures  that  are  reported 
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Measurements  that  Are  Reported 


Risks  identified 

156 

1461 

90.4% 

- 

Schedule  progress 

50 

1639 

97.0% 

- 

Code  growth 

490 

822 

62.7% 

- 

Defects  removed 

217 

1329 

86.0% 

- 

Defects  identified 

158 

1435 

90.0% 

- 

Effort  applied  to  tasks 

115 

1535 

93.0% 

- 

Capabi  1  ity/req  u  i  rem  en ts  stab  i  1  ity 

401 

1117 

73.6% 

0  200  400  600  800  1000  1200  1400  1600  1800 

Frequency 

1796  Responses 


Do  not  report 
Do  report 
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Summary  Observations  - 1 


In  general,  there  were  significant  differences  in  response 
patterns  when  comparing  management  versus  staff. 


Management 

Executive 
Program  Manager 
Project  Manager 


Staff 

Engineer 

Analyst 

Programmer 


Statistical  tests  of  significance  demonstrated  that  the  differences 
were  significant  with  confidence  of  at  least  99%  in  all  cases  and 
99.9%  in  some  cases. 

j  *  j  j  j  Hy  pothes  i s  tost  f  o  j  j  eq  u  a  I  ity  of  p  ropo  rti o n s  j 
3  3  3  3  3  Chi-Squar  e  te  st  for  sig  nif  icance  j  j  j  j  j  j  j  j  j  j  j  j  j  j  j  j  j 


©  2006  by  Carnegie  Mellon  University 


2006  State  of  SW  Measurement  Practice  -  35 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Summary  Observations  -  2 

When  compared  to  staff,  management  responded  more  strongly  that 

•  they  understand  the  purposes  for  measurement 

•  measurement  helps  their  team  perform  better  than  without  it 

•  they  use  measurement  more  often  to  understand  the  quality  of  their 
products  and  services 

•  they  follow  a  documented  process  more  often  for  collecting  and 
reporting  measurement  data 

•  measurement  definitions  are  commonly  understood  and  consistent  in 
their  organization 

•  measurable  criteria  exist  for  their  products  and  services 

•  corrective  action  is  taken  when  a  measurement-based  threshold  has 
been  exceed 

In  general,  the  differences  are  statistically  significant. 
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Summary  Observations  -  3 

It  is  notable  and  a  bit  alarming  that  only  40.3%  of  all  respondents 
reported  that  corrective  action  is  taken  when  a  measurement 
threshold  has  been  exceeded. 

Close  to  20%  of  respondents  reported  that  corrective  action  is 
rarely  or  never  taken  when  a  measurement  threshold  is  exceeded. 


Measurement  doesn’t  help  much 
unless  the  information  is  acted  upon. 
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Methods  Used 


The  CMMI  M&A  Process  Area  was  identified  as  a  measurement  method  used  most 
often  to  identify,  collect,  and  analyze  measurement  data. 

•  Approximately  56%  of  respondents  reported  using  the  CMMI  M&A  Process 
Area. 

•  27.4%  of  all  respondents  reported  that  the  CMMI  M&A  Process  Area  was  the 
only  method  that  they  used 

41%  of  all  respondents  stated  that  they  used  only  a  single  method  for  identifying, 
collecting,  and  analyzing  measurement  data.  (59%  use  two  ore  more  methods). 

A  21 .1%  (432  respondents)  reported  that  they  do  not  use  any  measurement 
method. 
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Summary  Observations  -  5 


Measures  Reported 


Schedule  and  time-on-task  measures  are  most  often  reported. 

•  97%  of  respondents  indicated  that  schedule  progress  was  a  measure 
most  often  reported. 

•  93%  indicated  that  effort  applied  to  task  was  reported. 

•  In  addition,  some  respondents  listed  other  measures 
that  they  report  and  19.2%  of  these  were  related  to 
time  tracking. 
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Summary  Observations  -  6 


Measures  Reported,  continued 


Code  growth  and  Capability  &  Requirements  Stability  are 
measurements  least  reported  by  respondents. 

•  27.3%  do  not  report  Code  Growth 

•  22.3%  do  not  report  Capability  &  Requirements  Stability 

Frequency  of  reporting  measurement  information  varied 
depending  on  the  measurement.  However,  most  are  reported  on 
a  weekly,  monthly,  or  daily  basis. 
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